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ABSTRACT 

Sight  is  the  most  important  sense  in  the  decision-making  process :  85%  of  the  information  we  receive 
comes  from  what  we  see. 

Traditionally,  simulators  aim  to  focus  on  reproducing  critical  situation  as  best  as  possible.  They  are  more 
focused  on  representing  the  3D  environment  when  moving  than  real  perception  of  the  scene.  Faithfully 
simulating  visual  perception  is  the  goal  of  the  technology  presented  in  this  paper. 

Thanks  to  two  research  and  development  projects,  in  simulators  and  in  realtime  visualization,  both 
applicable  in  virtual  reality  centres,  company  OPTIS  is  developing  an  advanced  true  visual  rendering 
technology,  developed  to  be  used  in  the  design  of  new  systems  and  for  training  purposes.  The  aim  of  the 
technology  is  to  reproduce,  on  a  display  and  in  a  virtual  reality  centre,  the  visual  perception  that  a  human 
would  have  of  a  scene  in  real  life. 

It  is  the  first  time  a  simulator  is  treating  all  lighting  effects  including  ultraviolet  and  infrared  band, 
encountered  in  a  scene.  Targeted  effects  include,  but  are  not  limited  to,  natural  lighting,  human  machine 
interfaces,  reflections  on  windshields  and  materials,  display  emission,  infrared  observation,  street 
lighting,  opponents  and  moving  targets. 


1.0  INTRODUCTION 

Designing  a  realtime  simulator  requires  high  speed  calculations  to  provide  images,  sometime  stereoscopic, 
to  the  operator.  Realistic  rendering  images  are  important  for  ensuring  that  the  trainee  behaves  authentically 
during  the  simulation:  85%  of  the  decision  is  taken  based  on  vision.  In  concrete  terms,  how  can  we  really 
take  a  decision  if  we  do  not  see  the  object  we  are  supposed  to  see,  because  of  the  sun  that  dazzles  you,  or 
simply  because  of  a  reflection  coming  from  a  street  light  around  you? 

As  in  many  domains  of  numerical  simulation,  having  realistic  behaviour  requires  adding  physics  models 
in  the  loop.  Light,  as  well  as  optical  simulation,  is  part  of  this  and  is  often  approximated  to  geometric 
perception  without  taking  into  account  the  energetic  balance.  Providing  a  correct  image  to  the  observer 
first  requires  a  correct  simulation  of  light  emission,  propagation  and  all  effects  that  will  not  only  be  in  the 
field  of  view  but  also  around  the  observer. 

Providing  realistic  images  means  going  one  step  further,  as  the  goal  is  to  model  the  way  the  light  will  be 
perceived.  It  therefore  requires  the  modelling  of  a  human  sense:  vision.  A  correct  model  of  the  human  eye 
requires  the  handling  of  linear  and  non  linear  functions,  since  human  vision  involves  multiple  sensors, 
with  variable  resolution,  and  effects  such  as  glare  which  affect  central  and  peripheral  vision. 

This  paper  describes  the  development  of  a  new  realtime  technology  integrating  both  light  effects  and 
human  vision  perception. 
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2.0  METHODOLOGY 

2.1  State  of  the  art 

Traditionally,  simulators  aim  to  focus  on  reproducing  critical  situation  as  best  as  possible.  They  are  more 
focussed  on  representing  the  3D  environment  when  moving  than  the  real  perception  of  the  scene  which  is 
the  goal  of  this  new  technology.  This  type  of  technology  is  strongly  based  on  graphics  shaders  or  the  using 
of  GPU  libraries,  which  offer  low  level  optical  models  of  light  sources  and  surface  qualities. 

In  driving  simulation,  some  systems  are  focused  on  night  driving  and  virtual  testing  of  headlamps  enabling 
you  to  simulate  the  effect  of  light  on  a  virtual  road.  They  provide  a  field  of  vision  of  the  lit  road  which  is 
displayed  on  a  large  screen  and  sometimes  several  displays  to  enlarge  the  driver’s  field  of  view. 

Adding  physics  to  simulators,  however,  requires  the  development  of  more  accurate  optical  models  both  for 
light  source  emission  and  for  light  diffusion.  This  means  it  is  necessary  to  rebuild  a  realtime  algorithm 
able  to  handle  full  light  propagation. 

Going  further,  it  is  necessary  to  take  into  account  human  vision  to  simulate  effects  that  projection  systems 
can't  reproduce  because  of  their  gamut  and  dynamic. 

2.2  Goal  of  the  project 

What  we  propose  with  this  new  approach  to  simulators  is  to  start  from  physical  data  to  achieve  a  final 
accurate  rendering  allowing  users  to  take  decisions  on  the  scene  they  are  seeing.  We  will  illustrate  this, 
using  different  types  of  simulators  in  day  and  night  conditions,  for  vehicles,  aircraft,  ship  and  land 
applications.  The  earlier  we  can  detect  a  danger,  the  better  it  is  for  the  design  process. 

The  goal  of  the  project  described  in  this  paper,  is  to  develop  a  new  simulator  technology  fully  based  on 
physics  and  the  physiological  perception  of  light,  taking  into  account  all  the  light  effects  which  occur 
during  operations.  This  simulator  technology  will  be  fully  compatible  with  virtual  reality  centres. 

2.3  A  physics  based  approach  to  rendering 
2.3.1  Energetic  consideration 

First,  to  make  an  accurate  rendering,  it  is  necessary  to  take  into  account  the  real  energy  in  the  scene.  This 
requires  spectral  simulation.  Most  of  the  current  real  time  simulators  use  a  RGB  spectrum  approximation. 
However  when  looking  at  the  rendering  equation  [1],  it  appears  that  reflected  radiance  computed  with 
spectrum  integration  is  different  to  the  radiance  computed  with  RGB  approximation: 
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•  0  is  the  angle  between  face  normal  and  considered  direction 


3-2 


RTO-MP-MSG-069 


_dP  NATO 
^  OTAN 


Light  and  Human  Vision  Based  Simulation  Technology 


•  cp  is  the  angle  between  face  binormal  and  considered  direction 

•  X  is  the  wavelength 

•  Lr  is  the  reflected  radiance  (W.sr-1  .m-2) 

•  Li  is  the  incoming  radiance 

•  p{6i , (p. ,Or,cpr,?C)  is  the  BRDF. 

In  the  night  simulation  with  street  lighting  for  example,  it  is  particularly  important  because  of  the  spectra 
singularity:  most  standard  lamp  used  for  street  lighting  and  vehicle  headlamps  are  emissive  gas  lamps 
which  are  usually  ray  spectra  (High  or  Low  Pressure  Mercury  or  Sodium,  Xenon,  ...).  LED  light  systems 
which  are  increasingly  used  today  also  present  a  relatively  thin  peak  in  the  spectrum. 

If  you  are  in  the  presence  of  an  object  absorbing  this  spectrum  ray  you  will  have  a  wrong  perception  and 
contrast  of  this  material  colour  and  reflectance  using  a  RGB  approach. 

Another  advantage  of  spectral  simulation  is  the  ability  to  simulate  near  infrared  camera  perception  and 
restitution.  Today’s  tactical  situations  require  the  use  of  such  a  system  to  detect  potential  dangers  arriving. 
The  camera  sensor  acquires  infrared  wavelengths  and  converts  it  into  an  electric  signal  in  order  to  display 
in  a  control  area  or  on  a  head  up  display  for  instance.  We  can  also  simulate  the  ultraviolet  part  of  the 
spectrum  to  enhance  the  visibility  of  white  objects  by  night. 

2.3.2  Light  source  emission 

All  light  sources  may  be  characterised  by  their  emission  spectrum  and  their  intensity  distribution.  Both 
characterise  the  way  the  light  source  emits  photons  anywhere  in  space.  The  intensity  distribution  is 
normalized  using  IES  format  and  is  provided  by  the  lighting  supplier. 


Figure  1:  Left  :  OSRAM  HP  Hg  Lamp  ray  spectrum  (visible  part)  -  Right :  intensity  distribution 


2.3.3  Light/matter  interaction 

Going  further,  to  be  physically  correct,  we  have  to  capture  the  spectral  response  of  each  surface 
reflectance  by  acquiring  its  real  Bidirectional  Reflectance  Distribution  Function  (BRDF).  This  can  be  done 
using  a  spectral  goniophotometer  and  the  portable  OMS2  system.  This  BRDF  has  to  fulfil  the  energy 
conservation  law  with  an  integral  <  1 .  This  basic  condition  is  not  usually  satisfied  with  standard  real  time 
BRDF  models  such  as  Phong  shading  model. 

These  measurements  have  been  done  for  different  types  of  material  inside  the  scene  (paint,  metal,  leather, 
apparel,  signs  ...)  to  enable  visibility  studies  to  be  carried  out. 
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Figure  2:  BRDF  function  represented  for  one  wavelength  /  one  incident  angle 

2.3.4  Light  measurement 

It  is  crucial  to  calculate  radiance  data  as  this  correspond  to  what  a  human  is  sensitive  to. 

Indeed,  the  quantity  of  light  coming  back  to  the  observer  depends  on  the  scene  material  (macadam, 
concrete)  and  if  the  surfaces  are  wet  or  dry.  The  perception  of  the  environment  can  greatly  differ 
depending  on  the  light  ambiance  too.  This  effect  is  due  to  the  fact  that  inside  the  eye,  light  is  diffused  by 
the  surfaces  and  media  encountered,  before  illuminating  the  retina.  It  is  described  by  [2]  [3]  [4].  This  stray 
light  decreases  sensitivity  that  may  prevent  the  observer  from  identifying  an  important  signal.  That’s  why 
our  algorithm  is  able  to  handle  a  high  variation  of  the  level  of  luminance,  like  moon  light  and  its  reflection 
on  the  road,  and  sunlight. 

In  a  vehicle,  interior  lighting  with  disturbing  dashboard  reflections  in  the  windshield  depends  on  the 
different  surfaces  involved  in  light  propagation.  As  specular  reflections  change  with  the  point  of  view,  it  is 
necessary  to  compute  them  in  real  time  and  allow  the  trainee  to  move  his  head  inside  the  vehicle  to  avoid 
disturbing  reflections.  LCD  displays  must  also  be  placed  carefully  inside  the  vehicle  in  order  to  avoid 
reflection  and  limit  the  glare  effect  on  pilots.  Head  up  displays  and  night  vision  goggles  can  be  located 
inside  the  vehicle  in  order  to  adapt  their  luminance  level.  For  all  these  cases,  the  choice  of  surface  BRDF 
model  and  reliable  physics  algorithms  is  crucial  in  order  to  have  an  accurate  estimation  of  the  nuisance 
caused  by  these  reflections.  Thanks  to  this  technology,  using  this  information,  even  designers  can  adapt 
interior  materials  and  colours  to  reduce  and  consequently  avoid  these  phenomena. 

The  entire  processing  phase  is  done  using  physical  units  (watt,  meter,  ...)  with  floating  point  precision, 
using  multi-core  Central  Processing  Unit  (CPU)  and  the  high  parallelism  that  the  Graphic  Processor  Unit 
(GPU)  offers  as  a  128  bit  SIMD  processor.  Working  on  floating  point  values  allows  the  production  of 
High  Dynamic  Range  (HDR)  spectral  images.  The  benefit  of  HDR  images  is  now  widely  recognized  for 
image  production.  The  latest  GPU  improvements  on  shaders  are  used  in  our  solution  for  colour  evaluation. 
Computations  are  done  at  the  pixel  level  to  take  into  account  sharp  (thin  gaussian  for  example)  BRDF 
effects.  For  each  source,  we  add  the  source  contribution  to  the  final  rendering  applying  the  corresponding 
BRDF  for  the  observer  direction,  the  considered  source  direction  and  spectrum.  Using  this  method,  we  are 
no  longer  limited  in  the  number  of  sources  whereas  OpenGL  is  usually  limited  to  eight  sources. 

2.4  Modelling  human  vision 

This  process  allows  us  to  take  into  account  the  HDR  image  perception  with  human  colour  perception.  A 
human  vision  model  was  developed  in  collaboration  with  worldwide  specialists  in  visual  perception  to 
transform  spectral  luminance  into  perceived  visual  information.  This  model  includes  a  day  /  night  vision 
mode  based  on  physiological  aspects,  and  takes  into  account  the  eye’s  response  to  light  levels,  colour, 
contrast,  as  well  as  rapid  changes  in  surrounding  luminosity.  The  human  eye  has  a  high  range  of 
luminance  detection,  able  to  detect  luminance  level  from  10"6  cd/m2  to  108  cd/m2.  Projection  systems  used 
in  a  simulator  don’t  have  such  a  high  dynamic.  So  it  is  necessary  to  develop  special  functions  better  able 


3-4 


RTO-MP-MSG-069 


Light  and  Human  Vision  Based  Simulation  Technology 


to  display  the  final  image,  as  if  the  scene  were  seen  by  a  human.  To  adapt  the  luminance  level  included  in 
the  HDR  image  to  the  screen  used  to  display  the  result,  we  perform  a  compression  taking  into  account 
human  vision  properties  and  namely  these  described  in  [5]  [6]. 

Sun  and  oncoming  lights  from  an  opponent  cause  glare  for  an  observer.  This  glare  reduces  observer’s 
perception  of  the  contrasts  of  the  scene  and  the  displayed  information.  Higher  intensity  light  around  the 
observer  (either  coming  from  the  surroundings  or  from  a  source  such  as  street  lighting)  reduces  glare 
experienced  by  the  observer,  as  the  contrast  between  ambient  and  oncoming  light  is  reduced. 


Without  alare 


With  alare 


Figure  3:  Simulation  of  glare  effect  due  to  sun.  Being  physics  based,  the  software  results  are  so 
reliable  that  measurements  can  be  taken  in  the  projected  image. 

In  a  vehicle  or  in  a  ship,  increasing  the  level  of  interior  lighting  causes  the  equipment  to  be  reflected  in  the 
windows  which  is  also  a  distraction  for  the  pilot-  something  designers  want  to  avoid.  By  simulating  the 
same  scene  using  different  interior  lighting  levels,  the  ergonomist  will  be  able  to  use  this  application  to 
find  the  best  compromise  between  glare  from  opponents,  and  reflections  of  the  equipment  in  the  windows. 
This  physiological  model  also  integrates  the  fact  that  the  optical  performance  of  the  human  eye 
progressively  declines  with  age  [7] [8].,  causing  loss  of  contrast  sensitivity  and  acuity,  increase  in 
discomfort  glare.  The  amount  of  light  scattered  into  the  different  media  of  the  eye  increases  with  age  [2]. 
Figures  4  illustrate  this  phenomenon. 
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Figure  4:  Simulation  of  the  increase  of  light  scattered  by  different  media  of  the  eye  when  driver 
age  increase.  Age  of  the  driver  is  20  on  left  image  and  60  on  right  image. 

This  technology  will  soon  take  into  account  the  temporal  adaptation  of  the  human  eye.  Indeed,  the  human 
visual  system  needs  a  release  time  before  reaching  the  final  stage  of  adaptation.  The  dark  adaptation  refers 
to  how  the  eye  recovers  its  sensitivity  in  the  dark  following  exposure  to  bright  lights.  This  type  of 
adaptation  is  observed  when  an  officer  enters  in  a  dark  control  room  after  receiving  glare  due  to  the 
sunlight.  The  light  adaptation  refers  to  how  the  eye  recovers  its  sensitivity  in  the  day  following  low  light 
exposure,  observed  for  example  when  an  officer  leaves  a  dark  area. 

At  the  end  of  the  simulation  process,  with  a  view  to  addressing  an  RGB  projection  system,  we  compute 
the  final  RGB  colour  from  the  resulting  spectrum.  This  computation  is  based  on  CIE  standard  observer 
functions  to  transform  spectrum  to  XYZ  colour  space.  This  last  step  takes  into  account  the  ICC  display 
profile  to  better  match  real  colours.  That  way  we  can  also  compensate  colour  variation  over  the  lamp’s 
lifetime. 

To  improve  the  user’s  immersion,  this  graphic  pipeline  can  be  duplicated  computing  two  different  images, 
one  for  each  eye  in  stereoscopic  mode,  using  head  tracking.  Using  an  accurate  BRDF  model  improves  the 
realism  as,  for  sharp  BRDF,  one  eye  can  see  a  disturbing  reflection  while  the  other  is  not  affected.  This 
reflection  behaviour  improves  the  surface  aspect  rendering,  which  is  the  only  information  to  address  visual 
perception. 


3.0  IMPLEMENTATION 

The  actual  implementation  of  our  simulator  is  based  on  Virtools.  Virtools  is  a  development  platform 
allowing  prototyping  and  industrial  applications,  it  also  manages  integration  with  Virtual  Reality  Centres 
such  as  CAVE-like  systems,  Reality  Centres,  Workbenches,  ...  and  also  manages  a  wide  range  of  VR 
devices  (trackers,  glasses,  haptic  systems,  ...).  Inside  Virtools,  the  OPTIS  rendering  part  is  completely 
separated  from  the  behaviour  part.  This  separation  allows  the  developer  to  manage  all  the  interactions  of 
the  trainee  with  his  environment  and  the  user  interface,  to  implement  his  own  lighting  strategy,  to  add  new 
geometry,  lights,  change  the  lighting  configuration,  ...  All  the  rendering  information  (sources  intensity 
diagrams,  source  spectrum  and  BRDF)  is  transferred  to  the  OPTIS  rendering  part  in  order  to  compute  the 
resulting  images  in  mono  and  stereo.  Input  files  (intensity  diagrams,  BRDF,  ...)  are  directly  loaded  into 
Virtools  environment  using  new  OPTIS  Building  Blocks  inserted  as  a  Virtools  plug-in. 
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4.0  APPLICATIONS 
4.1  Projectors  validation 

One  application  to  be  addressed  is  the  use  of  light  projectors  to  enhance  vision,  for  instance  helicopter 
trackers.  This  application  allows  the  evaluation  of  light  levels  in  a  city  and  represents  on  a  display  what 
the  pilot  can  really  observe  from  his  helicopter. 

This  also  allows  the  light  beam  to  be  checked  in  static  mode,  and  moreover  evaluate  illuminance  levels 
everywhere,  even  on  a  3D  building. 


Figure  5:  Irradiance  level  on  the  road  and  on  a  regulation  plane. 

Thanks  to  this  simulator  technology,  engineers  can  validate  tracking  light  systems.  The  orientation  of  light 
beams  can  be  automatically  calculated  according  to  the  position  and  movement  of  both  the  observer  and 
target.  Engineers  can  implement  this  tracking  function  inside  the  simulator  to  test  it  and  check  the  lighting 
properties  in  different  scenarios. 


Figure  6:  Light  simulation  (top  view).  Illuminance  levels  on  the  road  surface. 
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4.2  Security  and  safety  enhancements 

Enhancements  to  night  time  safety  and  security  are  a  goal  for  governments.  This  physics  based  simulator 
can  help  to  improve  safety  on  the  road  and  security  in  a  city  checking  new  equipments  such  as  safe 
jackets,  road  signs,  road  paints  ...  by  taking  into  account  the  real  behaviour  of  light  reflection  thanks  to  the 
use  of  an  accurate  BRDF  function. 


Figure  7:  light  reflectance  (BRDF)  applied  on  a  virtual  mannequin  to  check  its  visibility 

The  following  application  presents  a  3D  scene  illuminated  by  the  sun  (Fig.  8),  the  moon  (Fig.  9)  and  using 
night  vision  goggles  by  night  (Fig.  10). 


Figure  9:  night  scene  of  a  city  Figure  10:  same  scene  by  night  wearing  night  goggles 
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4.3  Optimize  the  ergonomy  of  a  cockpit 

As  described  previously,  this  technology  can  greatly  help  designers  and  ergonomists  in  their  tasks. 
Designers  can  directly  check  the  influence  of  surfaces  and  materials  inside  a  vehicle  and  their  behaviour  in 
different  lighting  conditions.  For  instance  common  Sodium  Lamp,  Mercury  Lamp,  Tungsten  reading 
lights,  Zenith  sun,...  will  all  produce  a  different  colour  effect  after  reflection  on  the  material  covering  the 
dashboard  or  on  the  instrument  panel.  The  designer  can  virtually  test  in  real  time  a  plastic,  leather,  and 
brushed  metal  from  his  library  and  select  the  most  appropriate  one.  This  selection  can  be  done  both  from 
an  aesthetical  point  of  view  to  enhance  perceived  quality  and  from  an  ergonomic  point  of  view  to  avoid 
unwanted  reflections  that  would  disturb  the  pilot.  It  is  particularly  the  case  in  night  driving  situations 
where  each  interior  light  can  produce  a  reflection  on  the  windshield  or  on  side  windows,  but  also  during 
the  day  in  the  case  of  clear  materials. 


Figure  11 :  the  cockpit  may  directly  come  from  a  3D  CAD  CAM  software  allowing  troops  to  be 
trained  on  a  system  which  does  not  yet  exist.  Left :  aircraft  -  Right  :  ship 
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Figure  12:  Different  light  configuration  can  be  selected  and  virtually  tested  during  training 


4.4  Performances 

The  current  implementation  of  our  simulator  in  a  case  of  ten  light  sources,  in  high  definition  images 
(1920x1024)  with  spectral  data  and  algorithm,  with  thirty  (30)  different  BRDF  files  on  an  NVidia  Quadro 
FX  5600  provide  a  twenty  frame  per  second  (20  fps)  application  reaching  performances  of  traditional 
simulators. 


5.0  CONCLUSION 

The  approach  to  vision  simulation  described  in  this  paper  is  a  new  technology  developed  to  improve 
simulators  by  displaying  visual  information  as  seen  in  real  life.  It  offers  an  understanding  of  the  visual 
perception  of  critical  situations,  allowing  us  to  better  train  people:  We  mustn’t  forget  that  a  human 
perfectly  detects  what  he  expects  to  see. 

Consequently  the  human  machine  interface  of  any  control  system  can  be  improved. 

The  photometric  simulation  part  has  been  fully  validated  over  more  than  ten  years  by  worldwide  users  in 
optronics,  lighting,  display,  automotive,  aerospace,  naval  and  consumer  goods  applications.  This 
development  was  certainly  a  key  to  obtaining  the  light  information  essential  for  vision  simulation.  The 
results  obtained  with  this  vision  simulation  have  been  compared  with  success  to  common  visual  tests  with 
a  view  to  being  validated. 

The  algorithm  used  enables  us  to  simultaneously  measure  the  impact  of  new  technology  lamps  on  the 
observer,  and  optimize  the  display  design,  the  position  and  orientation  of  the  display  in  a  vehicle,  taking 
into  account  all  light  effects  inside  the  vehicle  wherever  the  light  is  coming  from,  whether  inside  or 
outside. 

This  tool  will  also  enable  ergonomists  and  system  designers  to  improve  the  conditions  of  global  visibility, 
and  in  so  doing  the  global  perception  of  the  environment  that  the  observer  will  see  by  day  and  by  night, 
whatever  their  age.  Results  provided  include  luminance  (Cd/m2)  perceived  by  the  driver  allowing  also  to 
check  legibility  using  standard  as  CIE  145:2002  [9],  MIL-STD-1472F  or  ISO  150008:2003. 
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Figure  13:  OPTIS  landing  simulation  as  seen  from  the  control  room  with  realtime  glare  effects 
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Figure  14:  Detailed  view  of  glare  effects  showing  sparkling  and  diffraction  effects 


RTO-MP-MSG-069 


3-11 


Light  and  Human  Vision  Based  Simulation  Technology 


3-12 


RTO-MP-MSG-069 


